CS267 Project Proposal: Performance and Scalability Characteristics of Spark

In recent years, a new model of performing data-parallel computations on clusters of unreliable machines (e.g., MapReduce [1], Dryad [2]) has become widely popular. These systems achieve their scalability and fault tolerance by providing a programming model where users create acyclic data flow graphs to pass input data through a set of operators. This allows the underlying system to schedule jobs and to react to faults without user intervention. While this data flow programming model is useful for a large class of applications, applications that reuse a working set of data across multiple parallel operations cannot be expressed efficiently as acyclic data flows. Such iterative jobs are extremely prevalent in machine learning algorithms that repeatedly apply a function to the same dataset to optimize a parameter (e.g., through gradient descent). While each iteration can be expressed as a MapReduce/Dryad job, each job must reload the data from disk, incurring a significant performance penalty.